MacWorld 1998 October/Macworld (1998-10).dmg/Shareware World/Utilities/Text Processing/GST-TGhostDemo/Torquemada Explained/Torquemanual

Text File | 1998-08-01 | 79.9 KB | 1,892 lines | [TEXT/ttxt]

A Torquemanual for the Inquisitive




Documenting Torquemada version 1.3.1, © 1998 by Greg Swann


8/1/98

Greg Swann

[e-mail]
[e-mail]

USPS: 3608 West Cochise Drive
      Phoenix, AZ 85051




Table of Contents

0. Introductory Chatter

1. Torquemada Basics

2. String Literals

3. Aliases

4. Case Conversion

5. Untyped Wildcards

6. Typed Wildcards

7. The Do-It-Yourself Wildcard

8. Wildstrings

9. The Wildstring Modifier

10. Torquemada's Ghost

11. Putting It All Together

12. Bringing It All Back Home



0. Introductory Chatter

This is kind of a funny little Ugly Duckling story. Torquemada the
Inquisitor started life as a fluffy little bit of nothingstuff with
a Mac interface wrapped around it. When we were testing XP8, Garry
Fairbairn (the Argus of Saskatoon) had asked me to come up with a
search and replace engine that he could use to translate American
spellings to those favored by people more recently divorced from the
Crown. I came home one night and spent two minutes writing a little
toy that traded massive quantities of memory for very, very fast
searches. That little core was the basis of the first Torquemada.

Torquemada 1.0.0 worked only on string literals. It sported no
wildcards. It read only only one line at a time. But it was very,
very fast...

We've been through twelve versions since then. Along the way, the
search engine has been replaced twice, making it somewhat slower but
a lot less hungry for memory. The read/write code has been replaced
three times. The interface has been (transparently) overhauled. And
somewhere in there, I began to take it seriously... (grin)

The current release, version 1.3.1, can deploy a infinite number of
search and replace strings on up to 256 files in a drag and drop
batch, and the searches can employ up to 29 wildthings of various
stripes ("wildthings" are Torquemada's multifaceted wildcards; there
will be _much_ more about these in due course). And it is still
very, very fast...

And this is very far from the end of the story. But it is the end
for now, which makes this the appropriate place to fully document
the software as it now stands.


Commercial, legal and other pertinent notices:

XP8, mentioned above, is a text file reformatter. It will clean up
and make QuarkXPress-ready Macintosh or DOS text files. Among many
other features, it intelligently reformats paragraphs, converts the
DOS or WordStar character sets to their Mac equivalents,
substantially improves the hyphenation and justification of text,
converts quotes better than any software currently available, and
traps for XPress Tags errors that might otherwise result in missing
text or irreversible document corruption. A demo version of XP8 is
available on CompuServe (GO DTPFORUM, Library 5) or in the Info-Mac
archives (as GST-XP8Demo.sit).

This archive contains full or demo versions of Torquemada the
Inquisitor and Torquemada's Ghost, a scriptable, backgroundable
version of Torquemada. The demo versions will permit up to 32
launches and then forevermore refuse to do anything but be an
automated salesman. They are _fully functional_ demos. Nothing is
crippled, and everything works exactly as documented here. The only
inhibition is the one named: 32 launches, max.

All of these programs are sold on the same terms: (US)$50 each, per
license. Two to 10 licenses are $45 each. For 11 or more licenses
you're better off buying a site license. Information on registering
for the full releases of Torquemada and XP8, plus my other
commercial releases, can be found in the registration software
enclosed.

Torquemada the Inquisitor, Torquemada's Ghost, their source and
executable code, and this poor excuse for a manual are Copyright (C)
1998 by Greg Swann. All rights are most emphatically reserved.

Torquemada is licensed for use on one machine by the person who paid
for it. If you didn't pay for it, please do! I am one person, with a
long-suffering family, not Conglomerated MegaSoft (not to imply that
there's any virtue in ripping _them_ off!).

Torquemada is delivered "as is", without any warranties, expressed
or implied. It is not warranted to be useful _to_ anyone, _for_
anything, and in no wise am I to be held responsible for any
unfortunate consequences resulting from its use or misuse. And I
_hate_ having to say things like that. I do my best to write useful,
simple, elegant, bug-free solutions to difficult problems. If you
take it into your head that I represent your big chance to 'strike
it rich,' you will pay a lot in legal fees to discover that you have
miscalculated.

And: to those to whom the above disclaimer does not apply: forgive
me for having to make it. It's _you_ whom I'm working for, for pay
or for free. I appreciate your custom and your support, and I wish
we all could just comb the others out of our hair...

(Hey, it's a real 'personal' software company! (grin))


Some notes at random:

While XP8 is a QuarkXPress-specific program, Torquemada can be used
with any Macintosh software that can read and write files of type
TEXT (plain text files). For that reason, this manual is produced as
a plain text file. I would much prefer to write it as an XPress
file, but that wouldn't be terribly useful to people using
PageMaker, ReadySetGo! or FrameMaker. For ease of use with a variety
of text editors, this file is produced in DOS-like fashion; each
line ends with a carriage return. Ideally, you should read it and
print it out in a monospaced font such as Courier, since in certain
places things are aligned with spaces. And: where a certain
keystroke might be ambiguous, I am naming it in words enclosed in
braces; for example, {space} means the spacebar character.

In the same neighborhood, because I myself am an XPress user, and
because my own uses for Torquemada are focused on Quark, my examples
tend to be intensely Quark-like. This is a side-effect that will
have to be overcome by users of other software. Reasoning by
analogy, you will be able to see how to apply my examples to similar
features in your software of choice.

To make the best use of this software (and of XP8, for that matter),
you need some type of fairly functional text editor/word processor.
I use Word or BBEdit, a freeware programmer's editor. The latter is
highly recommended for our purposes.

As mentioned above, Torquemada owes its origins to some
exceptionally literate caterwauling by Garry Fairbairn. Its further
development was goaded in good measure by some very compelling
growling by Shane Stanley (who has a unique perspective on things
due to his insistence on living upside down in an entirely different
day). Along the way, a number of other users made valid, valuable
suggestions, and I'm sure that will continue to be the case. More
than any other piece of software by me, Torquemada has been
influenced by its users, and, here, at the outset, I want to
encourage you to contact me with any requests or suggestions you
have.

The default "Torquemada Prefs" file shipped with this disk pays
homage to the 20 brave souls who lent their diligence and
intelligence to the XP8 beta-testing process. Despite my gentle
ribbing, they have my gratitude and my highest respect.

Mike Arst (inventor of the Kvetchamatic Irregular Expression Parser)
had a profound and lasting impact on this document. Obviously, any
remaining errors or ambiguities are my responsibility; he takes
credit only for the stuff that is clear, grammatically correct, and
spelled properly in languages currently in use by humans. Seriously:
he has a rare talent for inducing a tabula rasa mental state (this
may not be a compliment!), such that he can spot hand-waving or other
logical elisions better than anyone I know. It was he who forced me
to write the (to me tedious) descriptions of what each and every
command means. If you are a software developer, you are well-advised
to pay Mike whatever he asks to critique your documentation. He can
be reached at [e-mail] or by USPS at:

Mike Arst
2459 Fifth Avenue West
Seattle, WA 98119-2506

Perhaps because of its democratic roots (how odd for a program named
after an autocrat), Torquemada the Inquisitor has sprouted a great
host of nicknames. I tend to call things by (quasi-)acronym, so he's
TQM to me. From the very beginning he was Torque or Tork to others,
and these gave rise to verb forms: I Torque, you were Torking, they
had Torqued.

And, for reference: Tomas de Torquemada was the First Grand
Inquisitor of the Spanish Inquisition. The name was later adopted as
a nom de guerre by the first editor of the Observer of London
Crossword Puzzle, and I probably would not have used it were it not
for that latter association.



1. Torquemada Basics

Torquemada is: 32-bit clean, System 7 compatible, Multi-Finder eager,
kind to children and house pets, and safe for use on your precious
hardwood floors. The companion software, Torquemada's Ghost, is
AppleEvent aware and is designed specifically to be used with
AppleScript or Frontier scripts.

Torquemada will run on any Mac from the Plus up. It requires 768K of
RAM, and you are cautioned not to set the partition smaller than
this; we're using it all.

The interface consists of two dialog boxes and two help windows, and
all of these are very straightforward in operation. If you can use a
Mac at all, you can use my boy Torque.

The Pyre of Purification dialog box uses 10 point Helvetica. If you
don't have that bitmap loaded, you will need to load it into either
your System file or your Core Fonts suitcase. If you don't,
Torquemada will still work, but the search and replace strings will
be hard to read.

And: when your files are being processed, Torque displays the stylish
and attractive Movado Museum Watch Cursor to let you know that things
are really happening.

TQM reads files of type 'TEXT'. It will also see files with the
creator types 'text', 'MDOS', 'mdos', 'CRLF' and 'crlf'. These latter
types are supported because some third-party products have very odd
ideas about how to treat 'TEXT' files.

Torquemada writes files of type 'TEXT' (with the creator type
'XP84'). These files can be opened from any application that can read
plain text files. They can be processed directly by XP8 or other
utilities of mine. Or they can be imported directly into QuarkXPress,
PageMaker or any other publishing application with a plain- or
smart-ASCII filter.

You write your search and replace strings into the Pyre of
Purification dialog box. Searches go on the left and replaces on the
right. Space is allotted for up to 20 search and replace strings, and
precious little space is left on a nine-inch screen! Torquemada
search and replace strings have an enormous maximum length (254
characters). The normal Macintosh text editing services are available
in the Pyre, along with Select All, Copy, Cut, Paste and Clear. You
can Copy from your text editor and Paste into the Pyre, which is the
fastest way to write search strings.

The Inquisitor reads and writes sets of search and replace strings.
This is valuable: in many cases, you need to perform the same searches
on the same types of files, week in and week out. Once you have
perfected a set of searches, you can save it for reuse. Through the
Open Set button in the Pyre of Purification dialog box, Torquemada
will read set files having the creator type 'PREF'. It will also 'see'
the types 'pref', 'TEXT', 'text', 'CRLF' and 'crlf'. These latter
types are supported so that, if you want, you can write sets in your
word processor. A TQM set saved from the Pyre of Purification consists
of 40 plain text lines in the form of search, replace, search,
replace, etc. When you hit the Save Set button in the dialog, the
software will write the currently active set to a file of type
'PREF'.

Note: if you don't intend to edit a set in the Pyre of Purification, it
can have an infinite number of search and replace strings. Also, a set
file can use any number of tabs as the delimiter between the search
and replace string. The reason is to permit easy editing of set files
in a text editor or word processor. Open the set file called "A Legal
Set File" from a text editor or word processor to see how this looks
in real life. Torquemada continues to Save sets using CR as the
delimiter. Be careful Pasting into WP docs; TQM filters tabs and
returns before he saves, but you'll need to do this manually after you
Paste. The Pyre of Purification will only edit files consisting of 20
search and replace strings. If you attempt to edit a set containing
more than 20 pairs of strings, you will get an ALRT inviting you to
Cancel! or Truncate the set.

If you have one basic set of searches that you frequently need to
run, you can save it under the name "Torquemada Prefs" and store it
in the same folder where TQM resides. When you launch the software
directly, this set will load automatically.

Alternatively, you can launch Torquemada by double-clicking on a set,
by selecting it and doing a Finder Open, or (under System 7) by
dragging the set onto the application's icon or an alias of it.
Torque will launch and the set you selected will be loaded instead of
Torquemada Prefs. For this to work, the set must have the creator
type 'PREF'.

You can select up to 32 sets (of type 'PREF') in this fashion, and the
sets will be processed in alphabetical order on your text files. These
sets will persist until you either Quit, Open Set, Save Set or Clear
All. After you do any of the latter three, only the set visible in the
Pyre of Purification will be run on your text.

IMPORTANT: Torquemada's Ghost processes set files in the order
received, not in alphabetical order. See the section on Torquemada's
Ghost for a fuller discussion of this important difference.

Under System 7, you can select up to 256 text files and drag and drop
them onto the application's icon or an alias of it. At the same time,
you can drag and drop up to 32 sets (of type 'PREF'). If you do not
select any sets, Torquemada Prefs will be run on your batch of files.
If you do select one or more sets, those sets will be run in
alphabetical order on each of your files.

To make this clear:

You can use the standard file interface to Open Sets and Open and
Save files; or,

You can select a number of sets in the finder and use the standard
file interface to have those sets run in alphabetical order on each
file Opened; or,

Under System 7, you can select a number of sets and a number of files
and drag and drop them all on Torque. Each file will be run against
all the sets.


Not for nerds only:

'Alphabetical order' is serious business. The 'natural' order is the
order selected (clicked upon) in the Finder. That's hard to remember,
and it's hard to predict the effects of multiple file selection done
by dragging out a marquee. So: we're sorting to lexical order: A
comes before Z, which comes before a, which comes before z. The files
'Today', 'TODAY', and 'today' would sort to:

TODAY
Today
today

O comes before o, and T comes before t. If you need to know how sets
will sort, just look at an ASCII chart. Better yet, name your sets
with numbers; use two digits (02, not 2), since 10 comes before 2.

Why are we sorting sets? A batch of sets is a 'virtual' set. The
second search in a single set can operate on the results of the first
search. In the same way, the first search in the second set of a
batch can operate on something left behind by the first set. To use
batches of sets reliably, you have to know the exact order in which
the searches will be made. Sorting the sets gives you greater
control.

REMEMBER: Torquemada's Ghost processes set files in the order received,
not in alphabetical order.

Why are the file limits set so high? Because I hate to think that
anything I do is _almost_ good enough. Honestly, if ever you find
yourself running hundreds of searches, examine your methods to see if
they can be made more general. If ever you find yourself running on a
batch of 256 files, query your source to see if they can tighten up on
their end a bit. But: if you really _do_ need awesome quantities of
strings or files, you've got 'em.

Batch-processed files will have the default extension (.TQM) applied
to them automatically. If a source file has already been Torqued, and
if its name is _exactly_ 31 characters in length, the new name would
be identical to the old. In that one case, we are using a slightly
different extension (.TQµ), so as to avoid toasting the source file.

Logic of Beeps: if you run a batch of files, Torquemada will beep at
you, a cheery little, "Hey, Dufus! Wake up!" TQM also beeps at the
end of any _normal_ run that takes longer than 5 seconds.


Nuts and bolts:

Torquemada reads and writes by the buffer-load. At read time, a
buffer is approximately 16K in size, and space is allocated for a
buffer to quadruple in size during processing. Torque can read around
returns _because_ it is working on these enormous buffers. At the
same time, we divide between buffers intelligently, so as to avoid
missing any searches.

At the time that a new buffer is assembled, a check is run against
the end of the buffer. If any of the search strings are present in
the end of the buffer, the buffer is truncated at the start of the
earliest full match. The remainder is prepended to the next buffer.

Each search string is run against the whole buffer _as it is at the
time of the search._ Consequently, later searches can operate on the
results of earlier replaces. This proves useful on a number of
grounds, as we will see in due course. But it is also grounds for a
certain amount of caution.

The buffer truncation mentioned above can only happen with strings
that represent the literal contents of the source file at read time.
So if a later search needs to find something put in by an earlier
replace, that earlier search needs to include everything expected by
the later search. As an example, this could fail ("^p" is a Torquemada
'alias' that denotes the carriage return character):

SEARCH STRING   REPLACE STRING
-------------   --------------
^pGreg          ^p<B>Greg<B>
^p^p<B>Greg<B>  ^p^p<BI>Greg<BI>

It could fail because the first search doesn't require Torquemada to
preserve both returns in the same buffer. In a very large file, with
a large number of chances to fail, some few can slip through the
cracks. The solution is simple: specify everything you'll need to see
in search strings that reflect the actual contents of the file:

^p^pGreg        ^p^p<B>Greg<B>
^p^p<B>Greg<B>  ^p^p<BI>Greg<BI>

Torquemada uses 29 characters in low ASCII for its own storage
purposes; all of the various wildthings are flagged with these codes.
In consequence, the software needs to insulate any low ASCII characters
found in the source. Otherwise, they would look like wildthings during
searches. So at read time, Torquemada is converting low ASCII found in
the source to a relatively harmless form, e.g. <\#008>. ASCII 1 though
8, 11, 12, and 14 through 29 are handled this way.

ASCII 10 (the linefeed) is handled with a certain measure of
intelligence. The linefeed is the poor relation among control
characters. It had a great day in the sun on Unix systems, where its
use is analogous to the carriage return on the Mac. On DOS systems,
it is an entirely redundant siamese twin to the carriage return. And
on Macs, the poor little linefeed isn't used at all. So: if
Torquemada finds himself processing a Unix file, he will convert the
linefeeds to carriage returns. Linefeeds in DOS files are thrown away
with malice aforethought.

ASCII 9 and 13 (Tab and Return) are not converted.

ASCII 30 is used by Word to denote non-breaking words, so we're
converting that to <\h>, the XPress Tags-language mnemonic for "don't
hyphenate". And Word uses ASCII 31 for the non-breaking hyphen, so
we're showing that as <\!#45>, XPress Tags for a non-breaking hyphen.
All of these conversions happen before any searches are run, so you
can search for the rare instance of low ASCII by using these codes.

On the way _out_ (as the new files is being written to disk) all
instances of the QuarkXPress literal code XPress Tag (e.g., "<\#000>")
will be converted to their actual ASCII value. This is implemented as
a post-process (i.e., after all searches are complete), so it is
possible to search for and act upon any tagged literal codes. Files
processed in this way can be run through Torquemada again
transparently, with the stipulations discussed above: ASCII 30 and 31
are permanently converted to <\h> and <\#45>, and LF (ASCII 10) is
always thrown away. If you introduce an LF into a file, be sure to
reintroduce it in subsequent uses of Torquemada.


User interface notes...

Torquemada 1.3.1 implements a number of new interface features. If
you're new to the software, follow along. But if you're not, follow
along _closely_. A _lot_ has changed.

First, when Torquemada is launched with a drag and drop batch of TEXT
files, it will quit automatically at the end of the batch. Drag &
drop batches of set files only will be handled as before, as will
double-click launches.

All of Torquemada's menu commands and buttons can be accessed via
keyboard equivalents. For example, the Wildcards scorecard can be had
with CMD-? from either dialog.

Any window or dialog with a close box can be closed with CMD-W.

In the Pyre of Purification (Strings) dialog, CMD-S will do a Save of
any _named_ file. If a file has not been named, that is, if the
program was launched by a double-click on its icon, Torquemada will do
a Save As instead. If you double-click on, drag and drop, or Open a
set, that is the set that will be Saved, and its name will appear in
the title bar of the dialog. If you double-click on or drag and drop
multiple sets, as described above, the set Saved will be the first,
lexically, and its name will be shown in the dialog. Note that when a
set is Saved or altered by any means, the rest of the sets in the
batch are forgotten about.

Important: the Save Set button works as before, when clicked upon. It
does a Save As, warning you if you try to save under the same name.

Also important: you can do a Save As on a named set by typing
CMD-OPT-S, a familiar enough keystroke. This is a quick way to create
multiple variations on a general theme.

After an Open Set or a Save Set (of whatever stripe), the name of the
currently opened set will be reflected in the dialog.

In the Pyre of Purification itself, we are showing the space character
as "˙" (ASCII 250, OPT-h) and the OPT-space character as "˚" (ASCII
251, OPT-k). This is done to make them visible, and therefore
_countable_. In consequence, if you actually need to search for either
of these characters (the literal "˙" or "˚"), you will need to 'alias'
them as "^˙" and "^˚". This seems a small price to pay for characters
you will (I promise) _never_ have occasion to search for.

There is more at work in the Pyre. We are exhaustively trapping the
keyboard. Only legal characters are passed (e.g., you can type CTRL-J
all day and nothing will happen), and some are intelligently parsed.
As above, space and OPT-space are converted as you type. If you
should happen to want to type a "˙" or a "˚", Torquemada will alias
either for you as you type (these characters will be shown unaliased
if you Paste them from another application; you're responsible for
your own aliasing there).

Certain Extended Keyboard keys are handled intelligently. Tab moves you
forward one field, and SHIFT-Tab moves you back one field. Up and Down
move you up and down one line, respectively. Home and Page Up take you
to the top of the current string. End and Page Down take you to the
bottom. The Delete key works (and this may be the lowest-priced
application in which it does!). Help brings up the Torquemada's
Wildcards scorecard, and SHIFT-Help presents the About Torquemada
window.

Best keyboard news: SHIFT-Left and SHIFT-Right select (highlight)
text in the way that most (non-nerd-oriented) applications work.

Torquemada's icons have changed. If you're a user of past versions of
the software, you may need to force the desktop to recognize that
there's a new Inquistor in town. To do this, first either delete or
archive all old versions of Torque. Then do this:

If you are running System Seven:

1. Select the program's icon and do Get Info (CMD-I)

2. Select the icon in the Get Info window

3. Copy, then Paste, then Cut

The Copy just loads the Paste buffer. The Paste replaces the
desktop's existing icon record for Torquemada with the contents of
the buffer. The Cut forces the desktop to update its records from
Torquemada's resource fork.

If you are running System Six:

1. Rebuild your desktop by restarting your Mac and holding down
CMD-OPT as it boots.

2. You will be advised that "this may take a while", and you will
find that "a while" is a unit of time greater than the lifespan of
some invertebrates.

After execution has begn, you can abort a batch of files with
CMD-Period. This will only happen between files, not between sets or
between searches.

And: there's some other stuff, but it's either transparent,
intuitively obvious, or an embarrassing correction to a long-standing
problem.



2. String Literals

The simplest use that can be made of Torquemada is to search for
string literals. This means literal text that actually appears in the
source file. For example, you could search for "Elvis" and replace
with "The King". If "Elvis" is present in the file, the replacement
will be made. Torquemada searches are case sensitive, which means
that if "elvis" or "ELVIS" are found in the file, the replacement
will not be made.

String literal searching is common; most applications that have a
search and replace function work only on string literals. Used this
way, TQM offers not much more than they, except that you can search
for 20 (or infinity!) string literals at once.

But consider: residents of British Commonwealth nations have a tough
way to go when they try to use text written in the United States. A
massive batch of string literals can be just the ticket.

And: string literals help to establish uniqueness in search strings
that employ wildthings of various stripes. (I'm not sure I really
have to say this, so I'd better.) Uniqueness means taking pains to be
sure that a search finds only those particular strings you want
changed, and not others. For example, our Queen's English-speaking
friends might want to change color and favor to colour and favour.
They might write the literal search:

or              our

This is pretty drastically non-unique. It would change the two
desired words, but it would also change the name of this software to
Tourquemada.

Uniqueness is one of those things that can only be learned the hard
way. So if the idea is new to you, spend half a day playing with
copies of files.



3. Aliases

ALIASES—Match special text characters
  ^T or ^t    Tab
  ^P or ^p    Carriage return
  ^^          Caret
  ^˙          ˙  ({OPT-h})
  ^˚          ˚  ({OPT-k})

You can't type a tab or a return in a dialog box. The Mac toolbox
filters for these characters, regarding them as commands. Tab moves
you from field to field, while hitting return is the same as clicking
the mouse on the Okay button. This is hardly news.

If you want to search for one of these characters, you have to use an
'alias' of it. Torquemada uses Word's convention for denoting these
aliases: tab is known as ^t or ^T, and return is wanted in five
states under the name ^p or ^P. The caret character is used to flag
all of Torque's many wildthings, so it too should be aliased in
search and replace strings: ^^. The caret is ideal for our purposes,
since it almost never appears in text. And, as discussed above, OPT-h
and OPT-k are aliased so that we can use them for better purposes.

Take note that you _can_ Paste tabs and returns into a dialog box. If
you do, Torquemada will behave as you expect (since the aliases are
just being turned into the literal tab and return characters at
search time). When you Save Set, a clean-up is run that converts any
Pasted tabs or returns into their aliased form.



4. Case Conversion

CASE CONVERSION COMMANDS—Can be used only on the replace side;
                         accented characters are handled
                         intelligently
  ^C or ^c    CONVERT TO ALL CAPS
  ^L or ^l    convert to all lower case
  ^S or ^s    Convert to sentence caps
  ^U or ^u    Convert To Upstyle Caps
  ^D or ^d    Convert to Downstyle Caps
  ^=          Cancel all case conversion

Torquemada includes six case conversion commands that are available
on the replace side only. If your source text is typed ALL CAPS, you
can convert it to sentence caps fairly easily. The commands can be
used only on the replace side because they control the output format
of the text found; they are not themselves something that can be
searched for. If you forget and put one in on the search side, TQM
will cheerfully ignore it with no untoward consequences.

But: case conversion by software is far from an exact science. The
commands will get you closer to where you want to be, but you may
have to run additional searches or do manual edits to achieve the
final desired results. This is particularly true of sentence caps and
downstyle caps (first letter of every word capitalized, except for
words of three or fewer letters).

However: Torquemada is pretty smart. If your text is totally toasted,
using these commands will save you a _lot_ of time. And the
Inquisitor is a good global citizen: accented characters are
converted appropriately. Moreover, Torque knows when to say when.
Quark's XPress Tags language is case-sensitive ('k' and 'K' mean two
different things). Only one of XP8's commands is case-sensitive. But
in both cases, we're insulating those commands from case conversion.

Face it: a file that is typed _all_ in the wrong case is the rare
bird. Further on, we'll see how to use these commands in conjunction
with other wildthings to selectively change case. For example, you
could take just the subheads to ALL CAPS, or change any ALL CAPS
headline to Upstyle Caps. For now, we'll just talk about using these
with string literals. This search:

Elvis           ^CElvis^=

would result in the word "ELVIS" being written out to the file. In
the replace string, ^C means convert to all caps, and ^= means cease
to convert case. (Of course, a search this simple could be done simply
with literals.) You don't need to use ^= if it is the last thing in
the replace string; Torque will put one there if it doesn't find one.
On the other hand, if you need to terminate case conversion _within_
the replace, then you must explicitly turn it off:

Elvis is still the king         ^CElvis^= is still ^Uthe king

would result in the text "ELVIS is still The King". We turned
capitalization off after "ELVIS" to avoid changing the case of "is
still". We didn't need to turn upstyle off, since it will be
terminated automatically at the end of the replace string.

(For users of earlier versions: Torquemada's case-conversion
routines were made substantially smarter with version 1.2.0.)



5. Untyped Wildcards

UNTYPED WILDCARDS—Match any one character
  ^0, ^1, ^2, ^3, ^4

Okay, this is where the bullet hits the bone. I've been dancing
around this for quite a while, because I wanted to establish good
ground rules before we got to the more hirsute topics. Well, here we
are...

Torquemada has 5 untyped wildcards. 'Wildcards' means they are not
literal characters, but rather markers that will match literal
characters. 'Untyped' means they will match _any_ character. In Word,
'?' is a wildcard, but it can only be used on the search side.
Torquemada's wildcards can be used on the search side, and can be
used, omitted or resequenced on the replace side. Torquemada's
untyped wildcards are denoted by the caret character followed by a
number from 0 to 4. Each one of them will match any character and
_store_ the character matched. Taking our earlier example:

^0^1^2or        ^0^1^2our

will match and change color and favor. Unfortunately, it still lacks
uniqueness. Because these wildcards are totally untyped, the search
string will also match any instance of the letters "or" and the three
characters before it.

Here's a better example. Suppose we have text that looks like this:

1. Blah.
2. Blah blah.
3. Blah. Blah blah.

If we needed to modify this, say to add a Quark 'Indent Here' command
to make sure that blahish turnovers hang on the indent, we could do:

^p^0.{space}    ^p^0.{space}<\i>

Surely we could do this with literals, but that would get tiresome.
Instead, we can use our knowledge to carefully control ignorance, and
accomplish in one string what might otherwise take three. Note that
we are isolating to uniqueness the text we hit; without the ^p and
the {space}, '^0.' would also match the ends of sentences.

Perhaps the best use of untyped wildcards is as ballast for the typed
wildcards (to be discussed next). They do what they do alone quite
well, but it's almost too much ignorance safely to be borne.

(Take note: prior to version 1.2.0, Torquemada had 10 untyped
wildcards. If you have old sets that made use of ^5 through
^9, you will need to revise them.)



6. Typed Wildcards

TYPED WILDCARDS—Match any one character of that type
  ^+    Uppercase character (includes accented characters)
  ^-    Lowercase character (includes accented characters)
  ^±    Character of either case (includes accented characters)
  ^&    Alphanumeric character (letter or number, not space or punctuation)
  ^%    Tabular character (digit, space or punct.; not alphabetical)
  ^$    Printable character (all characters _except_ space characters)
  ^¢    Any character _EXCEPT_ return
  ^!    Punctuation character (includes high-ASCII punctuation)
  ^.    Sentence Punctuation character (.,;:!?)
  ^#    Numeric character (digits only)
  ^_    Space character (space, return, tab, option space)
  ^¬    Space character (space, tab, option space, but _NOT_ return)

(^¢, ^. and ^¬ were added with version 1.2.0.)

These are alike unto the untyped wildcards except that they are
strongly typed. A typed wildcard will only match a character of its
type. So, for instance: ^+ will only match uppercase characters, with
all others failing to match. The wildcard ^# will match any digit,
and ^_ will match any space character, so we can go back to blahville
and do a much better job:

^p^#.^_         ^p^#.^_<\i>

This is now fully generalized yet completely unique. If there is a
subtopic such as:

a. Subblah.

it will fail to match. If we need to do something different with
that, we can match it with another, different string.

But: wildcards (typed or untyped) match and store only _one_
character. If you search for:

^#^#

only the second digit will be stored. If you need to match and store
two characters of one type, you can use the typed wildcards to
establish uniqueness, then fill out your team with untyped wildcards.
Suppose we needed to match:

10. Double-digit blah.

This would work:

^p^#^0.^_       ^p^#^0.^_<\i>

Pretty cryptic, not. This says: where you find a return followed by a
digit followed by any other character followed by a period followed
by a space character, soak it all up and spew it back out, appending
an XPress Tags 'Indent Here' command. We've gone from string literals
that were readable but not very useful to _this_, a vitally important
message from space aliens (grin). And it gets worse. Very shortly,
we'll discuss an even better - and more cryptic - way of handling
this type of problem.


Not for non-nerds only:

Mike Arst quite correctly pinned me to the mat for not going into
these guys in greater detail. Among the wildthings, these present the
greatest potential for confusion. Untyped wildcards (discussed above)
and wildstrings (discussed below) match _anything_. The typed
wildcards only match characters _of their type__ So: this is a
further elucidation of what "of their type" means.

Torquemada is written in the C programming language, and the idea of
typed wildcards is borrowed, analogically, from the character typing
functions available in the standard C function libraries. Where
appropriate, these commands use the full Macintosh character set,
where the C functions do not, but the idea is basically the same. A
typed wildcard matches the source character _if and only if_ the
character is of that type; ^+ will match _only_ uppercase letters,
not lowercase letters, not space or punctuation characters, not
digits.

Taking them one by one:

^+    Will match and store any one uppercase alphabetical
character. Alphabetical characters include the accented Macintosh
characters (e.g., Á, Å). This will _fail_ to match any character that
is not an uppercase character.

^-    Will match and store any one lowercase alphabetical
character. Alphabetical characters include the accented Macintosh
characters (e.g., á, å). This will _fail_ to match any character that
is not a lowercase character.

^±    Will match and store any one alphabetical character.
Alphabetical characters include the accented Macintosh characters
(e.g., Á, Å, á, å). This will _fail_ to match any character than is
not an alphabetical character. This wildcard is the logical opposite
of the tabular character wildcard (^%). (You type the "±" by
hitting SHIFT-OPT-=.)

^&    Will match and store any one alphanumeric character.
Alphanumeric characters are alphabetical characters (including
accented characters) and the ten digits. This wildcard will _fail_ to
match any space or punctuation character, which can make it useful
for establishing uniqueness.

^%    Will match and store any one tabular character. A tabular
character is any one of the ten digits, a space character (space,
return, tab, or option space), or a punctuation character (including
Macintosh high-ASCII punctuation such as ° or ‡). This wildcard is
intended primarily for matching the elements that make up a table,
exclusive of the explanatory text. Consequently, it will _fail_ to
match any alphabetical character, which can make is useful for
establishing uniqueness. This wildcard is the logical opposite of the
alphabetical character wildcard (^±).

^$    Will match and store any one printable character. A printable
character is one that makes marks on paper, which means that this
wildcard will match all alphabetical characters, all ten digits, and
all punctuation characters. It will _fail_ to match the space
characters (space, return, tab or option space). This wildcard is the
logical opposite of the space character wildcard (^_).

^¢    Will match and store any one character _that is NOT a carriage
return character_. Ergo, it will _fail_ to match return. (You type
the "¢" by hitting OPT-4.)

^!    Will match and store any one punctuation character, (including
Macintosh high-ASCII punctuation such as ° or ‡). It will _fail_ to
match alphabetical, numeric, or space characters.

^.    Will match and store any one sentence punctuation character.
The sentence punctuation characters are defined as: ".,;:!?" (not
including the dumb quotes). This wildcard was added because the
regular punctuation wildcard (^!) is too general in many
circumstances. This wildcard will match only the six sentence
punctuation characters and it will _fail_ to match all other
characters.

^#    Will match and store any one numeric character (the ten
digits). It will _fail_ to match alphabetical, space, or punctuation
characters.

^_    Will match and store any one space character (space, return,
tab, or option space). It will _fail_ to match all alphabetical,
numeric, or punctuation characters. This wildcard is the logical
opposite of the printable character wildcard (^$).

^¬    Will match and store any one space character _that is NOT a
carriage return character_. In consequence, it matches only space,
tab and OPT-space. It will _fail_ to match all other characters.



7. The Do-It-Yourself Wildcard

The Typed Wildcards are pretty through, but sometimes you just have
to Do-It-Yourself. This wildcard lets you. The DIY wildcard matches
the characters _you_ define, and it fails to match any you leave out.
Your definition can include literal characters (abc), ranges of
characters (a-c), or any of the other wildcards (^!abcx-z^#^_). Even
better, you can write the definition in the negative, specifying only
what will _fail_ to match. That is, you could define a DIY wildcard
that would fail to match all digits, all space characters, all
punctuation characters and all low-ASCII alphabetical characters;
this wildcard would match _only_ high-ASCII (accented) alphabetical
characters.

Let's ease our way into this, because it's a little complicated. The
form of the DIY wildcard is this:

^«...»

The leading caret is familiar enough by now. The next character, the
open-guillemot, is typed by hitting the OPT-\ (backslash) key. The
"..." represents the definition of the DIY wildcard, which we'll come
to shortly. The last character, the close-guillemot, is typed by
hitting SHIFT-OPT-\. This charcter tells Torque that the definition
is done. The definition of a DIY wildcard _must_ end with a "»", with
one exception. If the DIY wildcard is the very last thing in a
string, you can leave it out, in the same way that you can skip "^="
in similar circumstances. Everywhere else, the close-guillemot must
be present. If it isn't, the Inquisitor won't know where to stop the
difinition.

But what if you need a close-guillemot _as a part_ of your DIY
wildcard defiinition? Bright sparks will have already guessed the
answer: you alias it as ^». The string

^«^»»

will match _only_ the close-guillemot character.

On the search side, you show the full definition. On the replace
side, you show _only_ the ^«, omitting the definition. Like this:

^«abc»         ^«

The ^« is actually _storing_ the character found. The rest of the
difintion is just establishing _what_ will match, which is why it is
omitted on the replace side.

The definition of a DIY wildcard can be _retained_ from one string to
the next. If you put nothing between the opening and closing
guillemots, the previously-established definition will be used again.
As before, if it's the last thing in the string, you can leave off
the close-guillemot. So, for example:

^«abc»         ^«          (Matches a, b or c)
^«»            ^«          (Still matches a, b or c)
^«             ^«          (Still matches a, b or c)


Important: the DIY wildcard is just a special case of the ordinary
typed wildcard. It will match and store only _one_ character, the
first one found that fits the definition and the full pattern of the
search string. It behaves just like any other wildcard. If you use
more than one in the search string, only the _last_ one found will be
deployed in the replace. And, as with the other wildcards, it can be
used, omitted or resequenced on the replace side.

Okay, here's how to define one. This string:

^«ABCabc»

will match any one of those literal characters, and no others. You
can include as many literal characters as you want.

This string:

^«A-Z»

will match the _range_ of characters from A to Z, inclusive. A range
can be established by refrence to an ASCII chart. The range will
include every character from the start of the range to the end. For
example, you could do a less general tabular wildcard with

^«#-9»

This would match everyting from # (ASCII 35) to 9 (ASCII 57),
inclusive. (This is the definition XP8 uses for tabular characters).

Take note: in most GREPs, a range must be an _ascending_ range. This
is not true in Torquemada. You could just as easily say ^«Z-A» or
^«9-#».

You can have more than one range in a definition, or ranges plus
literals, like this:

^«A-Zz-a01234567890»

or this:

^«01234567890A-Zz-a»

or this:

^«A-Z01234567890z-a»

Each of these would match _only_ digits or low-ASCII alphabetical
characters, with all others failing to match.

And: you can include other wildcards _within_ your definition. This
string:

^«^#A-Zz-a»

does the same job as the three listed above. Since the DIY wildcard
is establishing the _type_ of characters it will match in its
definition, it makes sense to inlcude the other typed wildcards
within the definition. This:

^«A-Zz-a^#^!^_»

will match everything _except_ high-ASCII (accented) alphabetical
characters. Heck of a lot faster than typing all of those
hand-breakers.

Important: you can put either untyped wildcards or wildstring tokens
(discussed below) into your definition. If you do, the DIY wildcard
will match _anything_, since this is the range of characters defined
by those wildthings. I don't think this is such a swell idea, but I
made it possible just because you might need one extra untyped
wildcard someday. In any case, no matter what else is defined, a
(non-negative) DIY wildcard that includes either an untyped wildcard
or a wildstring token will match anything.

At the end of this section, we'll discuss some uses of typed
wildcards with the DIY wildcard.

Here's the coolest part: you can write a _negative_ definition. In
other words, you can define only what will _fail_ to match, with all
other characters matching. This string:

^«!A-Zz-a^#^!^_»

will match _only_ high-ASCII (accented) alphabetical characters. The
leading exclamation point establishes the definition as negative. The
remainder of the string specifies the characters that will _fail_ to
match the DIY wildcard. Since the only characters that are not
spaces, punctuation, digits or low-ASCII letters are accented
characters, these are the _only_ characters that will be matched.

As above, untyped wildcards and wildstring tokens can be included in
a negative definition, but the results are guaranteed to be
non-optimal. This

^«!^4»

says "match and store nothing". The "!" says "don't match" and the
"^4" says "every possible character". Search strings that include any
variation on this construct are entirely dysfunctional and will
result in no replacements. No extra charge...

The negative definition is a notational convenience. It does nothing
that cannot be done with a positive (normal) definition. It's just
faster, and, very often, easier on the hands and head. For example,
if you need a second ^¢, you can get it very easily with:

^«!^p»

But wait! What if we need a "!" as part of our definition? And what
about the "-" character? The exclamation point and hyphen establish
their meaning from their _position_ within the definition. The
definition is negative _if and ONLY if_ the exclamation point is the
first character in the definition. Otherwise, it's interpreted as a
literal text character. So if you need it in your definition, put it
anywhere but in the first position.

The hyphen is but slightly trickier. To include a hyphen in your
definition, put it in the first position, or in the second if the
first is occupied by an exclamation point. All other hyphens are
interpreted as defining ranges.

Here are some cool ways the DIY wildcard can be used along with other
wildcards.

If you have a font that uses the high-ASCII slots for purposes other
than those decreed by Adobe, you can use the DIY wildcard to insulate
those slots from case conversion:

^«A-z»          ^U^«^=

This would convert _only_ low-ASCII alphabetical characters to
Upstyle initial caps. There are some punctuation characters in that
range, but they don't matter since case conversion ignores them.
(And, yes,  both "»" and "^=" are redundant in this context, but Good
Practice decrees that we put them in. If, later, I add somehting to
the end of either string, I don't have to remember to add them, or
suffer the consequences of having forgotten to remember. In other
words, their omission is tolerated (and coded for) but Not
Recommended.)

You could use a very simple sort of definition to more firmly
establish uniqueness. For example,

^#^«^#»

will match _only_ two consecutive numbers. Something like ^#^& or ^#%
might let some wierdness through.

Obviously, you can use wildcards singly, as above, to create a
temporary twin. And you can use them in combination (along with
literals and ranges) to establish your own special types. And, of
course, the _antithesis_ of any typed wildcard can be had simply by
negating it. This

^«!^+»

will match anything that is _not_ an uppercase letter.

This is not the easiest stuff in the world, but it repays mastery. In
the section after next, you'll find out why we spent so much time
talking about it.



8. Wildstrings

WILDSTRINGS—Match and store any text until full pattern is matched
  ^*, ^~, ^?, ^@

Now this is the really cool stuff. There are four wildstrings and
each will match and store any text until the pattern defined in the
search string is satisfied. As an example, consider a full case
conversion. You got a file typed in ALL CAPS, and you need it to be
Sentence caps. This will do the job:

^~        ^S^~

The search string says: "match anything, from the start of the buffer
to the end." The replace string says, "spew it all back out with the
case converted".

A more complicated example: the client was thoughtful enough to
provide the keystrokes on disk, but the heads and subheads are all
typed ALL CAPS (there are many such thoughtful clients). To convert
them all to Upstyle (presuming that the heads to be hit are preceded
by double-returns), you could do:

^p^p^*^p        ^p^p^U^*^p^=

Wildstrings are massively general. Except for gross transformations,
you really have to build a lot of uniqueness into the pattern, or
they will bleed all over you. But try this on for size: I wish I
could count the number of times I've gotten files that were typed
like this:

First        Fifth
Second       Sixth
Third        Seventh
Fourth       Eighth

In every publishing system known to humanity, this should be typed as
one long column, with fifth following fourth. The helpful clients
type it in two columns with tabs so that you'll know it should be one
link spanning a gutter. Do it the cheap way, with tabs, and surely
you'll have to insert a line between second and third. Have fun
cutting and pasting... Or: run a set like this:

^t^*^p        ^p

This will throw away everything in the right column, preserving
everything in the left. Then run:

^p^*^t        ^p

This will throw away everything in the left column, preserving the
right.

Then concatenate the two files and you've got the text the way you
need it, in one long column.

How these work: wildstrings will match and store any characters found
in the source until the full pattern is matched. If the pattern is
matched immediately after the call to the wildstring, it will store
zero characters. This is useful. It means you can say:

^p^~^$        ^p^$

This will find all runs of one or more returns (or returns with other
space characters) and compact them down to exactly one return. If the
file has only single returns in it, the wildstring will contain zero
characters each time it is employed.

If the pattern fails to match, the text is passed through unaltered.

A wildstring can contain the whole buffer of text, as shown above.
Searching for a pattern match will continue to the end of the buffer.
In consequence, insufficient uniqueness can result in failures to
find a match along with very slow performance. So: be sure to deploy
these puppies in full cognizance of the actual contents of the file.

Wildstrings are completely untyped, but they can be fairly strongly
'typed' by the characters used before and after them. Needless to say,
string literals provide the strongest typing, but the more strident
typed wildcards also do very well. The string above that illustrates
how to compact runs of returns is a good example: we are saying
"match everything from a return to a printable character". Printable
characters are characters that are _not_ spaces, so the wildstring is
'typed' to contain _only_ space characters. A string like this:

^t^%^~^p        ^t<f"E Century">^%^~<f$>^p

will find the tabular columns of a table (viz., not the text stubs at
the left margin) and plug in XPress Tags coding to change the font.

A search string can contain more than one wildstring, up to the full
complement of four. At the same time, wildstrings can be resequenced
on the replace side. So, if that Officers and Directors chart comes
with the jobs before the names, you can swap them.

One more (this happened to me recently, and it's happened many times
in the past): pick up last year's financials, lose the right-most
column, kick everything right, and plug in a new column for this
year's figures. Bah! Mondo-beyondo manual labor, even with QuicKeys.
Here's a quick set that does the job on my kind of three-column
table:

<t41>^t<t-3>^t^*^t^~^t^?^p        <t41>^t<t-3>^tXX^t^*^t^~^p

The "XX" is there to hold the tab, to give the operator something to
double-click on. Yes, the operator had to type the figures;
intelligent hashing is for next week (grin). But: moving the columns
took minutes instead of hours, and the proofreader only had to read
the new copy, not the whole job.

Note: if you want, you can include the same one wildstring more than
once on the replace side. The set "Use last name twice" illustrates
the principle. If you run it on the file named "Use last name
twice.test", then compare the two files, you'll see how useful this
can be.

On the point: it has always been possible to repeat wildcards on the
replace side - just not terribly useful. Using wildstrings this way
won't come up every day, either, but it's a nice thing to have when
you need it.

There is a _ton_ of power in these babies, so do take the time to
master them, The effort will be repaid a hundred-fold.



9. The Wildstring Modifier

You may not know it yet, but _this_ is why you paid cash money for this
software. Every wildthing that was added with version 1.2.0 - ^¢, ^.,
^¬, and the DIY wildcard - was added in the service of this token. I
call it a token, because it's not really a wildcard. It doesn't
_match_ anything. What it does is constrain the type of characters
that a wildstring can match.

The wildstring modifier looks like this: "^<". You use it in strings
like this:

^#^<^~

That says: match and store one digit, then continue match digits
until the first non-numeric character, then _stop_ matching. In other
words, the wildstring will contain _nothing_ but (zero or more)
digits, nothing else. Its wildness has been _contstrained_, reined in
by by the wildstring modifier. An unmodified wildstring will match
and match and match, ceaselessly. But when it is preceded by the
wildstring modifier, it will _only_ match the type of the character
that immediately precedes the wildstring modifier.

In use, it looks like this:

^#^<^~          ^#^~

The wildstring modifier is omitted on the replace side, since its
only purpose is to constrain the wildstring during the search. If you
forget and put it on the replace side, guess what happens? Yes, of
course, Torquemada cheerfully ignores it.

The wildstring modifier is position dependent. It must be preceded by
a character or a wildcard (not a wildstring), and it must be followed
by a wildstring (not a character or a wildcard). If the sequence of
characters in the search string fails to satisfy that form, the
wildstring modifier itself is ignored (the rest of the search string
is honored).

If the wildstring modifier is preceded by a wildcard, the wildstring
is constrained to the type of that wildcard. For example, you could
clean up a badly-formatted table like this:

^%^<^?          |^%^?          (Mark all table columns)
^_^<^?|         ^t             (Compact all space runs before
                               table columns to one tab)

And yes, the wildstring modifier can be legally preceded by an
untyped wildcard, but this would be an entirely redundant
construction.

If the wildstring modifier is preceded by a literal character, the
wildstring is constrained to _that character EXACTLY_. Earlier we
talked about using wildstrings to compact runs of returns (keep in
mind that "^p" is an alias for a literal character). Here's a far
better way:

^p^<^?          ^p

Here's a similar example: the client made a fill-in-the-blanks form
using scads of underbars. You could do this:

_^<^?           ^t          (Convert underbar runs to a tab)

Of course, if you want to retain a certain minimum number of
underbars, just use one more than that number in your search string.
This will preserve three or fewer:

____^<^?        ^t

_Only_ the character or wildcard immediately preceding the wildstring
modifier is used to constrain the wildstring. Suppose you need to
"cast" the wildstring to more than one type...?

If you said "DIY wildcard", score one for you. Of course, the _whole_
pattern must match before the replace will take place, as with the
four underbars above. But you can keep it short and sweet with the
DIY wildcard. The table example above can let some stuff through that
you might not want, resulting in superfluous tabs. This is more
unique, hence safer:

^«#-9»^<^?          |^«^?          (Mark all table columns)
^¬^<^?|             ^t             (Compact all space runs before
                                   table columns to one tab)

The first search defines "tabular character" more rigorously
(^«$%(),.-^#» is more rigorous still), resulting in fewer false
positives. The second line matches _only_ space characters _that are
in the same paragraph_ as the mark we left behind. This is _why_ both
"^¬" and "^¢" exist. One of Torquemada's best features is his ability
to read around returns, but sometimes it's too much of a good thing;
you can wait a long time to achieve results far removed from your
expectations. Torquemada does _exactly_ what you tell him to do.
Those two typed wildcards, in conjunction with the wildstring
modifier, let you order the Inquisitor around in more strident terms.
So, what's the benefit? In the second line above, the string is
forbidden to toast the return in source copy that looks like this:

Stub     $123
Stub     $456
         $789

That stubless third line would get hosed by our first example, but
not the second.

The trick is to manage your knowledge in such a way that you can use
it to leverage ignorance into certainty. In a badly formatted table,
there will be _plenty_ of extra spaces, but only _some_ of them
should be compacted to tabs. The rest should become single spaces.
We're using our knowledge to isolate cases of the first type, then
operating intelligently on _only_ those. The marking trick we're
doing in the table will be discussed more in the next section.

This has come up: the text looks something like this:

1 This line contains K1.
2 This line contains K1 and K2.
3 This line contains K2.
4 This line contains K2 and K1.
5 This line contains K1.
6 This line contains K1 and K2.
7 This line contains K2.
8 This line contains K2 and K1.

where "K1" and "K2" are known instances of literal text. We want to
operate _only_ on lines that contain both K1 and K2, in that order.
This example is obviously trivialized; in real life we can't count on
the consistency of the surrounding text. A straight wildstring search

K1^?K2        <B>K1^?K2<B>

would result in garbage. It would also take a while on a big file,
since any failures will have to fail to the end of the buffer. No
fault: Torquemada is doing what we told him to do. By contrast,

K1^¢^<^?K2        <B>K1^¢^?K2<B>

will do exactly what we want, and very quickly to boot. Because the
wildstring can only match characters that are _not_ the carriage
return character, the search pattern will _fail_ to match as soon as
that character is encountered. Only lines 2 and 6 match the pattern,
so the replace string will be deployed only on those two.

And: you can use DIY wildcard definitions to create additional
same-paragraph wildstring modifiers. For example

^«!^!^p»^<^?

will match only non-punctuation characters that are all in the same
paragraph.

Because "^<" stores nothing, there is no restriction on the number of
times it can be used on the search side. Obviously, you should not use
it any more than four times (one for each wildstring) if you are
retaining the copy from all wildstrings. But if you are tossing copy
for one or more wildstrings, you can use it as many times as you need it.

For example:

If you have copy like this:

Good    junk    junk    junk    junk    junk    Good    Good

this string:

^?^t^$^<^*^t^$^<^*^t^$^<^*^t^$^<^*^t^$^<^*^t^~^p     ^?^t^~^p

will pack and repack all of the junk into the same one wildstring,
which is being thrown away in any case.

Included on this disk is a commented set called "Extract Text from
PostScript". This is still under construction. There is much about it
that is not ideal, just yet. It permits too much garbage to pass
through, and it takes a _long_ time to run (since it's main job is
matching and throwing away everything that _isn't_ text). But it gets
better every time I play with it, and it illustrates some of these
concepts very well. And - who knows? Someday you may need desperately
to recover the keystrokes from a PostScript file. With the wildstring
modifier, you can do it...



10. Torquemada's Ghost

Torquemada's Ghost is simply an AppleEvent-aware version of the
Torquemada S&R engine. While it can be used interactively by Drag &
Drop, its primary purpose is to allow AppleEvent scripters to access
Torquemada from their AppleScript or Frontier scripts. Frontier
scripter Eugene Barnes has lent us his expertise by writing Frontier
glue scripts for Torquemada's Ghost. These scripts and their
documentation can be found in the folder "Torquemada’s Ghost Glue ƒ".

In essence (I am not a scripter), you would access Torquemada's Ghost
from a script like this: send a host of set files and text files to be
processed to the Ghost. As the last file in the batch, send a 'control
file', a text file with the creator TQM+. The file "GhostBuster"
explains how to build a control file (it ain't hard).

The control file exists to tell the Ghost what to do with the files
it processes. It can contain a 'path' to a destination folder. If it
does, then the newly created '.TQM' files will be created in that
destination folder, rather than in the same folders as the source
files.

When we finish a batch started by a control file, we make
an empty file called "Batch completed..." in the home folder of the
control file. That way, your script can keep trying to delete that
file until it succeeds. While it is failing, the batch is not
complete. As soon as it succeeds, the batch is done. In C, the loop
would look like this:

    while (FSDelete ("\pBatch completed...", controlFile.vRefNum))
        ;

FSDelete is going to return a (non-zero) error message when it fails,
so we just do this until we get a zero back, indicating success. (Note
that I wrote this in C because that's what I know. Pursue Eugene's
examples to find out how to do it in Frontier.)

It is also possible to use Torquemada's Ghost interactively, by Drag &
Drop. Just drop your sets and files on the icon, much as you would
with Torquemada the Inquisitor. When you hit "Start", the files will
be processed as you would expect - taking account of the important
differences noted below.

Interactively, it is also possible to 'target' files to a specified
destination folder. If you hold down the Command key as you hit
"Start...", you will be prompted for a destination folder for your
newly-created files.

Whether by script or double-click, if Torquemada's Ghost is launched
without receiving any files, it will stay resident until it is
explicitly quit. If it is launched _with_ files, it will quit at the
end of the batch. As with Torquemada the Inquisitor, you can abort a
batch of files with CMD-Period. This will only happen between files,
not between sets or between searches.

IMPORTANT DIFFERENCES: First, while Torquemada the Inquisitor
processes set files in alphabetical order, Torquemada's Ghost
processes S&R set files IN THE ORDER RECEIVED. The reason is this: set
files in Torquemada's Ghost can originate from many different folders.
Presumably, some are boilerplate sets used to solve common problems.
Imposing an alphabetization restriction on them would surely drive
everyone crazy.

Second, text files, too, can originate from different folders. You can
Drag & Drop from all over, and everything will go back where it came
from - or, alternatively, where you specify - when you hit "Start".

Third, Torquemada's Ghost can run in the background. If you have a
huge job to do, you can set it up and start it running, then go do
other work. Torquemada's Ghost uses a lot of cycles, so backgrounding
is not as terrifically useful as it might at first sound, but it _is_
possible. Note that even in the foreground, Torquemada's Ghost will be
somewhat slower than Torquemada's the Inquisitor.



11. Putting It All Together

There is a boatload of Torquemada sets travelling with this document,
to illustrate various points discussed. None of the sets here is very
elaborate, but my feeling is (despite what you may surmise from
reading the DTPForum on CompuServe) the elaborate set is the
exception, rather than the rule. Most search and replace jobs are
fairly simple if they can be generalized to their essence.

As an admonition: if this stuff is largely new to you, you are well
advised to take your Torque in small doses. Build the vicious set
that does the worst of the reformatting, then have a look at the
output file. Write a new set to do the finer sifting, then have
another look. There is no shame in having a file named
"file.TQM.TQM.TQM.TQM". The shameful thing would be toasting your
text by trying to do too much at once. My own files tend to be named
"kill.XP8.TQM.XP8.TQM", since I use XP8 to do the gross clean up,
Torque to code for XP8, XP8 to unpack the Torque coding (e.g., I'd
much rather replace with "[n77]" than "<f"Univ NewswCommPi">M<f$>"),
then Torque, finally, to finesse. What works best is what works
fastest, not the set that wins the Nobel Prize for Cryptic
Communication With Space Aliens (this from the man who came up with
all these cryptic commands!).

Here's a brief discussion of the sets enclosed and what they do:

'PC to Mac PostScript' uses literals and one wildstring to clean up
PostScript files that originate on DOS machines but are being
downloaded from Macs. Torquemada automatically removes the linefeeds
(which are not a problem in any case). This set removes other
characters, common in PC-PostScript files, which _can_ be a problem
(most notably the control-D character which starts and ends many
PC-PostScript files).

'Stupefaction' uses nothing but string literals and aliases to recode
Mac-like text into a form that can be used on computers less swift.
Ideally, you should reformat the output in an editor (e.g., Save As
Text Only With Linebreaks from Word). That way, those DOS pigs might
actually be able to open the file.

'Reformat DOS File (Commented)' does the opposite. It takes files
that originated on DOS systems and reformats them to a fairly
Mac-like form. This set is interesting on several grounds. First, it
illustrates how to comment a TQM set: if the search string is empty,
the replace string is ignored. Consequently, you can embed
explanatory comments in your sets simply by typing them into the
replace side. Since you can run up to 32 sets in a batch, this can be
a swell idea: logically separate the types of searches you're
running, then comment each set for future reference. This file is
also making moderately interesting use of wildstrings. And finally,
the set is using markers to permit intelligent processing of special
cases.

This warrants its own paragraph: recall that a later search can look
for things left behind by an earlier replace. It is entirely possible
to have a problem complicated enough that you cannot fully resolve
all doubts in one search. In a case like that, you can leave markers
behind (viz., "|" or "][", anything that is unlikely to show up in
the text), then operate on them with later searches. In extreme
cases, you might need to drop in two or more markers, then operate
intelligently on the quantity present when you get around to ditching
them.

Or, suppose you get a file in Atari or Commodore ASCII. This seems
unlikely at this late date, but stranger things have happened. The
ASCII in those two systems was swapped: uppercase lived where
lowercase belonged, and vice versa. Presumably this was done for a
reason (to make sorting difficult?), but, whyever, it's a big problem
for you. This set will toggle the case of any file:

^+        |^+
          Mark existing caps
^-        ^C^-
          Existing lower case to ALL CAPS
|^+       ^l^+
          Marked caps to lower case

'Sentence Caps' shows the first thrust at a very thorny problem.
Suppose you have a file that was typed ALL CAPS. Not all that common,
but it happens. Getting to Sentence caps is fairly easy, but in
getting there, you will have lost most of the caps on proper nouns.
Get set to run a load of literals, because that's the only way to
catch most of them. But at least one can be captured with a
generalized search, as shown in the second search in this set. This
will convert names in the form of Firstname I. Lastname back to
initial caps.

'XP8 to PageMaker®' takes text processed by XP8 or Saved As XPress
Tags from Quark and puts it into a form that can be used by the Smart
ASCII filter in PageMaker. Aldus has promised a smarter Smart ASCII
filter. If you use PM, you might entreat them to hurry, because there
is a lot of cool stuff that can be done with Torquemada if there is
proper support from the destination application.

'Preserve Left Column' and 'Preserve Right Column' are discussed
above.

'Code Alternating Paragraphs' uses wildstrings to code any file that
comes in the form of:

A
B
A
B

Examples of this type of file: Q&A files, Officers and Directors
tables, phone or store listings, etc. The codes shown are XPress
Tags, but this would work just as well with PageMaker Smart ASCII
tags.

'Code Well-Ordered File' uses wildstrings to code files that come in
the form of:

Head
Sub
Body
Sub
Body
Head
Body
Head
Sub
Body

The presumption is that headlines and subheads are preceded by
multiple returns, which is usually the case with this type of file.
Note that we are using two different body styles, since frequently
you want to omit the paragraph indent for the first paragraph after a
head.

And you can see the point: the judicious use of Torquemada can remove
much of the labor - and certainly the most onerous labor - from a text
processing job. In my own work, I am striving to do everything with
software, with no manual labor at all...



12. Bringing It All Back Home

This is way awesome cool, if I do say so myself (grin).

My little ugly duckling has become a text-processing powerhouse.
Infinite searches on 256 files with 29 wildthings yields a
solution to all but the most intractable text-processing problems...



Very Best!,

Greg Swann
8/1/98



APPENDIX A - TORQUEMADA QUICK REFERENCE


ALIASES—Match special text characters
  ^T or ^t    Tab
  ^P or ^p    Carriage return
  ^^          Caret
  ^˙          ˙  ({OPT-h})
  ^˚          ˚  ({OPT-k})

UNTYPED WILDCARDS—Match any one character
  ^0, ^1, ^2, ^3, ^4

TYPED WILDCARDS—Match any one character of that type
  ^+    Uppercase character (includes accented characters)
  ^-    Lowercase character (includes accented characters)
  ^±    Character of either case (includes accented characters)
  ^&    Alphanumeric character (letter or number, not space or punctuation)
  ^%    Tabular character (digit, space or punct.; not alphabetical)
  ^$    Printable character (all characters _except_ space characters)
  ^¢    Any character _EXCEPT_ return
  ^!    Punctuation character (includes high-ASCII punctuation)
  ^.    Sentence Punctuation character (.,;:!?)
  ^#    Numeric character (digits only)
  ^_    Space character (space, return, tab, option space)
  ^¬    Space character (space, tab, option space, but _NOT_ return)

DO-IT-YOURSELF WILDCARD-Match any one character of the type you define
  ^«...»    "..." is your definition

WILDSTRINGS—Match and store any text until full pattern is matched
  ^*, ^~, ^?, ^@

THE WILDSTRING MODIFIER-Constrain following wildstring to type of
                        preceding wildcard or character
  ^<

CASE CONVERSION COMMANDS—Can be used only on the replace side;
                         accented characters are handled
                         intelligently
  ^C or ^c    CONVERT TO ALL CAPS
  ^L or ^l    convert to all lower case
  ^S or ^s    Convert to sentence caps
  ^U or ^u    Convert To Upstyle Caps
  ^D or ^d    Convert to Downstyle Caps
  ^=          Cancel all case conversion





APPENDIX B - A SERMON ON TOLERATION


As I say elsewhere in this manual, Torquemada The Inquisitor owes his
name to a joke originally played by an exceptionally crafty crafter
of crossword puzzles. But, upon reflection, I've decide that I can't
make a joke of that kind without a word or two of explanation.

For the year 1992 marked not only the 500th anniversary of the
discovery by Columbus of the West Indies, but also the 500th
anniversary of the expulsion of the Jews from Spain by the original
Torquemada, Frey Tomás de Torquemada, Dominican friar, Inquisitor
General of The Holy Office of Spain - more popularly known as The
Spanish Inquisition.

The former event is being met with studied indifference punctuated by
occasional breast-beating by people who can never hope to match
Columbus in stature, either in virtue or vice. The latter event is
all but forgotten, to our peril.

Pity the post-modern West! What spiteful, envious, small-minded
gossips we have become. We absolve ourselves of sloth and complacency
by trying to sully every manifestation of the truly heroic we can
find. We can safely excuse ourselves of striving to the virtues of
Columbus by focusing on his vices. We can forgive ourselves for
lacking Ferdinand's courage - the courage that drove the Moorish
empire from Granada - by nattering about his bastards. We can muddle
along in the fog of the safely irresolute, with our only firm
conviction being that the conviction of Isabella - who stared down
the armed chivalry of the Spains - is "impractical". After all, she
had to pawn her jewels to fund that fool's errand of the Genoese...

We are a "practical" people, and we set no value above our safety,
our comfort, and our leisure. Heroes are demanding, so we have idols
instead. We demote heroes at the first sight of a flaw, thus sparing
ourselves the demands of the heroic. For our idols, anything goes,
but of our heroes we will tolerate no imperfection. The West allows
but one symbol of heroism to stand, albeit on one leg. And, of
course, he allowed his feet to be washed by a whore, a juicy tidbit
we do well to keep from Torquemada's spiritual heirs...

Thankfully, we are "practical" enough to have "practical" political
parties, the Gutless Socialists and the Gutless Facists, going by
different names in different nations. Thankfully, because by virtue
of their lack of both courage and conviction, they are not apt to do
anything too terribly ambitious, such as build crematoria. We have
made ourselves "practically" safe, all without having to stray too
far from the sofa and the remote control...

Why, we're "practically" as safe as were the Israelites in the
newly-formed nation of Spain!

"Reasoning by specious analogy!", we can protest, conveniently
ignoring the blood of 150 million political murders that has already
stained this century. "Demagoguery!", we can shout, neglecting the
re-enslavement of the entire globe on terms far more rapacious than
those demanded by the Romans or the Huns or the Moors or the Goths.
"But we are modern!", we can cry. Modern enough to convict Columbus
and acquit Stalin. Modern enough to mourn the slaughter of the
indigenous Americans and the Maori but to forget entirely the
slaughter of the Kulaks, of the Jews, of the Gypsies, of the Chinese,
the Koreans, the Vietnamese and the Kampucheans. We are very modern
indeed. That primitive Torquemada could only murder innocent people
one at a time...

In Europe, Africa and Asia, they know it _can_ happen "here". In the
United States, it's awfully easy to forget that it _has_. The vicious
practices of Senator Joseph McCarthy were taken chapter and verse
from methods perfected in the late fifteenth century by Tomás de
Torquemada - though, being ignorant, McCarthy probably borrowed them
from the Communists he was excoriating.

And bear in mind, while our leaders are largely gutless, their
supporters are not. Whether in the name of "Decency" or "Political
Correctness", there is never any shortage of people with the
conviction that "infidels" with unpopular ideas should not be
suffered to exist and the courage to back that conviction by force of
arms.

"Decency" is fine thing. So are "Political Correctness" and
"Traditional Family Values" and "Respect For Alternative Lifestyles".
All fine ideas. But when we attempt to "persuade" others of their
value with a gun - or with a law, which is the Modern Couch-Potato's
Remote Control Gun - we have left entirely the realm of civilized
discourse and entered instead the jungle of barbarism and tyranny.
What starts as rowdiness and vandalism, if unchecked, ends at an
_auto de Fe_, at a forced march to the sea, at Treblinka...

Consider the irony: Torquemada killed people for refusing to renounce
their ideas in the name of a man who died rather than renounce his
ideas! I am not a Christian, but I find much to admire in the heroism
of the Nazarene. And nothing at all to admire in either Torquemada or
his So Very Modern successors. Our prized safety, our vaunted
comfort, our empty leisure - we owe these things to people who were
neither safe nor comfortable nor idle, who valued nothing ahead of
the convictions of their own minds, who feared nothing in the glare
of that bright-burning light. As we bring slander and torture and
exile and murder upon them, we bring poverty and tyranny upon
ourselves. This is the lesson that Ferdinand and Isabella learned too
late. If we cannot be _principled_, one may hope we are at least
smart enough to be "practical".

We would not be human if we did not seek desquite from ideas that
make us uneasy. But before we "Cry Havoc!", we would do well to see
just what kind of dogs we are unleashing. That man from Nazareth, the
last hero we suffer to stand, might be able to wash away our sins. It
seems unlikely that he can wash the blood of innocents from
Torquemada's hands. Or from our own...




APPENDIX C - BLATANT ADVERTISING

These are my currently shipping commercial products:

XP8 - a very intelligent file filter that cleans up and makes the
filthiest text QuarkXPress-ready. Among many other features, it offers
DOS-file reformatting, financial-text clean-up, garbage disposal,
typographic quality enhancement, and the best quote conversion we know
of. The demoware version of XP8 can be found under the name
XP8.SEA or in the Info-Mac archives as GST-XP8Demo.sit.

Torquemada The Inquisitor - batch global search and replace software
with wildcards, pattern matching, string substitution, et very
cetera. With Drag & Drop under Systems 7 or 8, you can run up to 640
searches on up to 128 files in one batch. Features the most
intelligent case-conversion we know of. The most-recent FreeWare
version (1.1.0) can be found under the name TORQUE.SEA in Library 5
or in the Info-Mac archives as GST-TorqueDemo.sit. The current
commercial version is 1.3.0, offering a great many enhancements,
including new "wildthings" and a _lot_ of new User Interface power.
The commercial version ships with Torquemada's Ghost, a scriptable,
backgroundable Torquemada. A DemoWare version of Torquemada's Ghost
is available as TGHOST.SEA in Library 5 or in the Info-Mac archives
as GST-TGhostDemo.sit.

Shane the Plane 2.0.2 - file and font attribute editing utility.
Interactively or in Drag & Drop batches, permits you to change the
Creator/Type of files, their created/modified dates and times, a
host of significant Finder flags, plus a lot more. Makes files
invisible/visible, makes fonts behave like files by removing their
BNDL resources, batch "pastes" custom icons, intelligently renames
and/or "slugs" files, et very cetera. A demonstration version (fully
functional but limited to 32 launches) can be found in Library 12
under the name SPDEMO.SEA or in the Info-Mac archives as
GSU-STPDemo.sit.

Mark My Words - a very elaborate MS-Word binary to QuarkXPress Tags
text filter. It eats Word 4.0, 5.0 or 5.1 files, interactively or by
Drag & Drop, and converts the binary to QuarkXPress tagged text. You
can elect to include or omit any feature of Word's styling, and many
features can be converted from their WP-like form to their DTP-like
form (e.g., underscoring to italic). With Em Software's Xtags
Xtension, picture and text boxes (including Word's tables) can be
retained. A demonstration version (fully functional but limited to
32 launches) can be found in Library 12 under the name MMWDEM.SEA or
in the Info-Mac archives as GST-MMWDemo.sit.

(While I've vectored all the files toward CIS and the internet, my
primary haunt, they are also available on other services, and on any BBS
which has the most recent version of AMUG's BBS-In-A-Box CD-ROM on
line.)

All of these programs are sold on the same terms: (US)$50 each, per
license. Two to 10 licenses are $45 each. For 11 or more licenses you're
better off buying a site license. All of this is explained in the
registration software supplied with this archive.